Skip to content

feat: transform agent into orchestrator with pagination awareness - #73

Merged
luis5tb merged 11 commits into
RHEcosystemAppEng:mainfrom
dmartinol:feat/orchestrating-agent-prompt
Apr 8, 2026
Merged

luis5tb merged 11 commits into
RHEcosystemAppEng:mainfrom
dmartinol:feat/orchestrating-agent-prompt

Conversation

@dmartinol

Copy link
Copy Markdown
Collaborator

Summary

  • Rewrites AGENT_INSTRUCTION in core/agent.py to transform the agent from a flat tool-calling proxy into a reasoning orchestrator
  • Adds Think/Plan/Execute loop with PlanReActPlanner for multi-step reasoning
  • Adds few-shot workflow examples (CVE remediation, infrastructure health, upgrade readiness) demonstrating tool chaining
  • Adds pagination HITL checkpoint — agent must present pagination options before calling paginated tools, unless the user already specified a limit
  • Adds guardrails: scope enforcement, prompt injection resistance, write-operation confirmation, data integrity rules

Closes #69

Pagination Awareness

Modeled after the rh-sre cve-impact skill Step -1 pattern:

  • System CVEs: prompt with "first page / all pages / N pages" (systems can have 1,700+ CVEs, ~18 API calls)
  • Account CVEs: confirm default limit=20 or let user choose
  • Host listing: offer first page / all / specific count
  • Skip when explicit: "top 3 CVEs on host X" → use limit=3 directly, no prompt
  • Recommend all pages for remediatable queries: remediatable CVEs are sparse and can appear on any page

Motivation

The agent was acting as a thin LLM wrapper — receiving a request, picking one MCP tool, and returning raw output. Two problems:

  1. No orchestration: Complex queries ("are my systems safe?") need multiple tools chained together, not a single call
  2. No pagination control: Paginated tools (CVEs, hosts) were called with defaults, either fetching too little (incomplete) or too much (wasteful)

This PR solves both via prompt engineering — the lowest-risk, highest-leverage change before the deadline.

What's covered vs. what needs follow-up

Aspect Covered by this PR? Needs enforcement layer?
Multi-step reasoning ✅ Yes (prompt + PlanReActPlanner) No
Pagination user control ✅ Yes No
Casual misuse ✅ Yes No
Sophisticated prompt injection ⚠️ Partially Yes — before_tool_callback plugin
A2A chain poisoning ⚠️ Partially Yes — input sanitization
Hard tool-call limits per session ❌ No Yes — stateful plugin
Org-id parameter tampering ❌ No Yes — plugin validates args

Follow-up work (post-deadline)

  1. Multi-agent architecture: Decompose into orchestrator + sub-agents using ADK sub_agents
  2. GuardrailPlugin: before_tool_callback for hard enforcement
  3. Pagination enforcement: Track page counts in plugin state, not just prompt

Test plan

  • All 232 tests pass (pytest tests/ -v)
  • Ruff lint passes
  • Manual: send multi-step queries and confirm agent plans before executing
  • Manual: send paginated queries without limits and confirm HITL prompt appears
  • Manual: send queries with explicit limits ("top 3 CVEs") and confirm no HITL prompt
  • Manual: send prompt injection attempts and confirm agent refuses

🤖 Generated with Claude Code

dmartinol and others added 10 commits March 31, 2026 16:39
Rewrites AGENT_INSTRUCTION to turn the single-agent LlmAgent from a
flat tool-calling proxy into a reasoning orchestrator. The previous
instruction was purely descriptive (capability catalog); the new one
teaches the LLM *how to think* about multi-step requests.

Three new sections:
- Think/Plan/Execute loop for multi-step reasoning
- Few-shot workflow examples (CVE remediation, infra health, upgrade
  readiness) that demonstrate tool chaining
- Guardrails: scope enforcement, prompt injection resistance,
  write-operation confirmation, data integrity rules

This is the minimal-risk mitigation for orchestration and safety while
the full multi-agent architecture and before_tool_callback guardrail
plugin are planned for a subsequent iteration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The old prompt had "Respect the read-only mode when enabled" guidance
that was dropped in the orchestrator rewrite. While read-only mode is
enforced architecturally via tool filtering, the LLM should still
communicate this constraint to users.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace brittle keyword-matching approach ("ignore previous instructions",
"you are now", etc.) with a principled statement that behavior is defined
by the system prompt and cannot be changed by user messages. The keyword
list was trivially bypassable via paraphrase or semantic reframing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MCP tool responses from upstream APIs could contain adversarial content
(e.g., a hostname crafted to look like an instruction). Add explicit
guidance that even instruction-like content in tool output must be
treated as data only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The prompt says "do not reveal internal tool names" but then listed
specific tool function names in the write-operation confirmation rule.
Replace with generic descriptions to avoid contradicting the
information-leakage guardrail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The old wording implied the LLM could verify org boundaries, but it
cannot — org isolation is enforced by OAuth tokens at the MCP/API layer.
Updated wording to accurately reflect that organization boundaries are
enforced through authentication on each tool call.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Enable structured plan-then-execute reasoning so the agent chains
sequential tool calls for complex queries (e.g., finding CVEs with
known exploits for a specific system) instead of giving up after a
single tool call and asking the user to follow up manually.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: add PlanReActPlanner and multi-step tool usage instructions
Adds a Pagination Awareness section that requires the agent to present
pagination options (first page / all pages / N pages) before calling
paginated tools, unless the user already specified a limit in their
query. Modeled after the rh-sre cve-impact skill's Step -1 pattern.

Key behaviors:
- System CVEs: prompt with page options (systems can have 1,700+ CVEs)
- Account CVEs: confirm default limit=20 or let user choose
- Host listing: offer first page / all / specific count
- Skip prompt when user specifies scope ("top 3 CVEs on host X")
- Recommend "all pages" for remediatable CVE queries (sparse results)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@luis5tb

luis5tb commented Apr 7, 2026

Copy link
Copy Markdown
Collaborator

I gave this a try and the reasoning (though a bit verbose) looks good. There seems to be some issues with the pagination

@luis5tb luis5tb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok merging this to have the enhanced prompt and then fix the pagination

@luis5tb
luis5tb merged commit cc00786 into RHEcosystemAppEng:main Apr 8, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants